Calibration Of Portable Devices In A Shared Virtual Space

ABSTRACT

Methods, systems, and computer programs for generating an interactive space, viewable through at least a first and a second handheld devices, are presented. The method includes an operation for taking an image with a camera in the first device. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the taken image to identify a geometry of the second device. Furthermore, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.

CLAIM OF PRIORITY

This application is a Continuation application under 35 USC §120 andclaims priority from U.S. application Ser. No. 12/973,827 entitled“Calibration of Portable devices in a Shared Virtual Space,” and filedon Dec. 20, 2010, which claims priority from U.S. Provisional PatentApplication No. 61/311,251, filed Mar. 5, 2010, and entitled“MAINTAINING MULTIPLE VIEWS ON A SHARED STABLE VIRTUAL SPACE”; and U.S.Provisional Patent Application No. 61/323,762, filed Apr. 13, 2010, andentitled “CALIBRATION OF PORTABLE DEVICES IN A SHARED VIRTUAL SPACE”,all of which are incorporated herein by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. application Ser. No. 12/647,291(Attorney Docket No. SONYP095A) filed Dec. 24, 2009, and entitled“WIRELESS DEVICE PAIRING METHODS”; U.S. application Ser. No. 12/647,296(Attorney Docket No. SONYP095B) filed Dec. 24, 2009, and entitled“WIRELESS DEVICE PAIRING AND GROUPING METHODS”; U.S. application Ser.No. 12/647,299 (Attorney Docket No. SONYP095C) filed Dec. 24, 2009, andentitled “WIRELESS DEVICE MULTIMEDIA FEED SWITCHING”; and U.S.Provisional Application 61/319,021 (Attorney Docket No. SONYP109+),filed Mar. 30, 2010, and entitled “METHOD FOR AN AUGMENTED REALITYCHARACTER TO MAINTAIN AND EXHIBIT AWARENESS OF AN OBSERVER”, which areincorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to methods, devices, and computer programsfor controlling a view of a virtual scene with a portable device, andmore particularly, methods, devices, and computer programs for enablingmultiplayer interaction in a virtual or augmented reality.

2. Description of the Related Art

Virtual reality (VR) is a computer-simulated environment, whether thatenvironment is a simulation of the real world or an imaginary world,where users can interact with a virtual environment or a virtualartifact either through the use of standard input devices or specializedmultidirectional input devices. The simulated environment can be similarto the real world, for example, simulations for pilot or combattraining, or it can differ significantly from reality, as in VR games.Virtual Reality is often used to describe a wide variety ofapplications, commonly associated with its immersive, highly visual,three-dimensional (3D) environments. The development of Computer AidedDesign (CAD) software, graphics hardware acceleration, head-mounteddisplays, database gloves, and miniaturization have helped popularizethe notion.

Augmented Reality (AR) provides a live view of a physical real-worldenvironment whose elements are merged with (or augmented by) virtualcomputer-generated imagery to create a mixed reality. The augmentationis conventionally in real-time and in semantic context withenvironmental elements, such as sports scores on television during amatch. With the help of advanced AR technology (e.g. adding computervision and object recognition) the information about the surroundingreal world of the user becomes interactive and digitally usable.

The term Augmented Virtuality (AV) is also used in the virtual realityworld and is similar to AR. Augmented Virtuality also refers to themerging of real world objects into virtual worlds. As an intermediatecase in the Virtuality Continuum, AV refers to predominantly virtualspaces, where physical elements, e.g. physical objects or people, aredynamically integrated into, and can interact with the virtual world inreal-time. The term VR is used in this application as a generic termthat also encompasses AR and AV, unless otherwise specified.

VR games typically required a large amount of computer resources.Implementation in handheld devices of VR games is rare and the existinggames are rather simplistic with rudimentary VR effects. Additionally,multiplayer AR games allow for the interaction of players in a virtualworld, but the interactions are limited to objects manipulated by theplayer in the virtual world (e.g., cars, rackets, balls, etc.) Thevirtual world is computer generated and independent of the location ofthe players and the portable devices. The relative location of theplayers with respect to each other and with respect to theirsurroundings is not taken into account when creating a “realistic”virtual reality experience.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present invention provide methods, systems, andcomputer programs for generating an interactive space viewable throughat least a first and a second handheld devices are presented. It shouldbe appreciated that the present invention can be implemented in numerousways, such as a process, an apparatus, a system, a device or a method ona computer readable medium. Several inventive embodiments of the presentinvention are described below.

In one embodiment, a method includes an operation for taking an imagewith a camera in the first device. In addition, the method includes anoperation for determining a relative position of the second device withreference to the first device, based on image analysis of the takenimage to identify a geometry of the second device. Furthermore, themethod includes operations for identifying a reference point in athree-dimensional (3D) space based on the relative position, and forgenerating views of an interactive scene in corresponding displays ofthe first device and the second device. The interactive scene is tied tothe reference point and includes virtual objects, and each view showsall or part of the interactive scene as observed from a current locationof the corresponding device.

In another embodiment, a method, for generating an interactive spaceviewable through at least a first device and a second device, includesan operation for taking an image with a camera in the first device,where the first device and the second device are handheld devices. Inaddition, the method includes an operation for determining a relativeposition of the second device with reference to the first device basedon image analysis of the image to identify a light source in the seconddevice. Further yet, the method includes operations for identifying areference point in a three-dimensional (3D) space based on the relativeposition, and for generating views of an interactive scene incorresponding displays of the first device and the second device. Theinteractive scene is tied to the reference point and includes virtualobjects, and each view shows all or part of the interactive scene asobserved from a current location of the corresponding device.

In yet another embodiment, a method, for generating an interactive spaceviewable through at least a first device and a second device, includesan operation for taking an image with a camera in the first device,where the first device and the second device are handheld devices. Inaddition, the method includes an operation for determining a relativeposition of the second device with reference to the first device, basedon image analysis of the image to identify a graphic shown on a seconddisplay of the second device. Additionally, the method includesoperations for identifying a reference point in a three-dimensional (3D)space based on the relative position, and for generating views of aninteractive scene in corresponding displays of the first device and thesecond device. The interactive scene is tied to the reference point andincludes virtual objects, and each view shows all or part of theinteractive scene as observed from a current location of thecorresponding device.

Other aspects of the invention will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts a user before synchronization a portable device to areference point in space, according to one embodiment.

FIG. 2 illustrates a virtual reality scene observed with the portabledevice.

FIG. 3 illustrates how movement of the portable device has a similareffect on the display as when moving a camera in the virtual space,according to one embodiment.

FIG. 4 shows a two-dimensional representation of the change in the imageshown in the display when turning the portable device, according to oneembodiment.

FIG. 5 illustrates how to play an interactive game over a networkconnection, according to one embodiment.

FIG. 6 illustrates the process of tapping two portable devices tosynchronize their positions, according to one embodiment.

FIG. 7 shows two portable devices after tapping, according to oneembodiment.

FIG. 8 illustrates the creation of a virtual scene around the referencepoint, in accordance with one embodiment of the invention.

FIG. 9 depicts two players in the same space where a virtual reality hasbeen created around a reference point on a table, according to oneembodiment.

FIGS. 10A-10B illustrate the process of detecting a location of anotherportable device using image recognition, according to one embodiment.

FIG. 11 shows how to detect the second device by finding a light sourcein the second device, according to one embodiment.

FIG. 12 illustrates detecting the second device by finding the displayof the second device, according to one embodiment.

FIG. 13 shows an embodiment for tracking the portable device via deadreckoning.

FIG. 14 illustrates how to adjust dead reckoning using static featuresin the background, according to one embodiment.

FIG. 15 illustrates one embodiment of a calibration method for amulti-player environment.

FIG. 16 depicts a multi-player virtual reality game, according to oneembodiment.

FIG. 17 shows the flow of an algorithm for generating an interactivespace viewable through at least a first and a second device, inaccordance with one embodiment of the invention.

FIG. 18 illustrates the architecture of a device that may be used toimplement embodiments of the invention.

FIG. 19 is an exemplary illustration of scene A through scene E withrespective user A through user E interacting with game clients 1102 thatare connected to server processing via the internet, in accordance withone embodiment of the present invention.

FIG. 20 illustrates an embodiment of an Information Service Providerarchitecture.

DETAILED DESCRIPTION

The following embodiments describe methods, apparatus, and computerprograms for generating an interactive space viewable through at least afirst and a second device. It will be obvious, however, to one skilledin the art, that the present invention may be practiced without some orall of these specific details. In other instances, well known processoperations have not been described in detail in order not tounnecessarily obscure the present invention.

FIG. 1 depicts a user before synchronization a portable device to areference point in space, according to one embodiment. Portable device102 is standing on a table in preparation for synchronizing the portabledevice to a reference point. User 102 has placed the portable device ina point that will serve as a reference point or anchor to build avirtual reality around the point. In the case shown in FIG. 1, theportable device is sitting in the approximate center of a table, and avirtual world is built around the center of the table once the portabledevice is synchronized. The portable device can be synchronized in avariety of ways, such as pushing a button on portable device 104,touching the touch-sensitive screen in the portable device, letting thedevice stand still for a period of time (e.g., five seconds), entering avoice command, etc.

Once the portable device receives the input to be synchronized, positiontracking modules in the portable device are reset. The portable devicecan include a variety of position tracking modules, as discussed belowin reference to FIG. 18, such as an accelerometer, a magnetometer, aGlobal Positioning System (GPS) device, a camera, a depth camera, acompass, a gyroscope, etc.

The portable device can be one of many types, such as a handheldportable gaming device, a cell phone, a tablet, a notebook, a netbook, aPersonal Digital Assistant (PDA), etc. Embodiments of the invention aredescribed in reference to a portable gaming device, but the principlescan be applied to any portable electronic device with a display.Principles of the invention can also be applied to game controllers orother input devices connected to a computing device with a display.

FIG. 2 illustrates a virtual reality scene observed with the portabledevice. After synchronizing device 104 with respect to reference points106, the portable device will start displaying a view of the virtualreality 108. The view in the display is created by simulating that acamera in the back of the portable device moves within the 3D spacearound reference point 106. FIG. 2 depicts a virtual reality thatincludes a chess board. Portable device 104 is capable of detectingmotion and determining its relative position with respect to referencepoint 106 as the device moves around. Location and positiondetermination can be done with different methods and different levels ofaccuracy. For example, location can be detected by analyzing imagescaptured with a camera, or data obtained from inertial systems, GPS,ultrasonic triangulation, WiFi communications, dead reckoning (DR),etc., or a combination thereof.

When the user enters a command to set the reference points, all motionsensing devices are zeroed out or calibrated to that position in space.For example, the user can place the device on the table and press abutton to calibrate all motion sensing data (accelerometers, gyroscope,GPS, etc.) From this point on, all captured positional information forthe device is recorded and processed relative to the initial calibratedposition via inertial navigation tracking. All subsequent capturedpositional information is considered relative to the device calibratedposition (reference point.)

In one embodiment, the device keeps track of the location in space ofthe portable device with respect to reference point 106, as well as theposition in space of the portable device. The position is used todetermine the viewing angle of the camera, that is, the portable deviceacts as a camera into the virtual scene. If the portable device is aimedtowards the right, then the view will turn to the right, etc. In otherwords, the viewing angle is defined as a vector with origin in thecenter of the display (or other part of the device), and with adirection perpendicular to and away from the display. In anotherembodiment, only the position in space is tracked, and the view in thedisplay is calculated as if the camera is aiming from the location inspace where the portable device is located and towards the referencepoint.

In some existing implementations, an AR tag is placed on a table, andutilized as a fiduciary marker for generating the augmented reality. TheAR tag may be an object or figure that is recognized when present in thecaptured image stream of the real environment. The AR tag serves as afiduciary marker which enables determination of a location within thereal environment. Embodiments to the invention eliminate the need for ARTags, because of the calibration within the 3D space and the locationtracking of the portable device. Additionally, the location informationallows games in the portable device to deliver a realistic 3D virtualexperience. Further, an array of networked portable devices can be usedto create a shared virtual world, as described below in reference toFIGS. 15 and 16.

FIG. 3 illustrates how movement of the portable device has a similareffect on the display as when moving a camera in the virtual space,according to one embodiment. FIG. 3 shows car 302 inside a virtualsphere. Assuming that a portable device is aimed from a point in thesphere towards car 302, multiple views of the car can be obtained as theportable device moves within the sphere. For example, a view from the“north pole” will show the roof of the car, and a view from the “southpole” will show the bottom of the car. Also shown in FIG. 3 are viewsfor the side, front, and rear of the car.

In one embodiment, the player can enter a command to change or flip theview of the virtual world. For example, in the case of the car, a playergoes from seeing the front of the car to seeing the back of the car, asif the scene had rotated 180° around and axis running vertically throughthe reference point. This way, the player does not have to move aroundthe room to get different viewing angles. Other inputs may producedifferent effects, such as a 90° turn, a scaling of the view (to makethe virtual world seem smaller or greater), rotation with respect to thex, y, or z axis, etc. In another embodiment, a flip of the portabledevice, i.e., a 180° spin on the player's hand will cause view of thevirtual world to flip upside down.

FIG. 4 shows a two-dimensional representation of the change in the imageshown in the display when turning the portable device, according to oneembodiment. Portable device 402 is aimed towards a wall with a viewingangle α, resulting in a projection 410 on the wall. Thus, the view onportable device 402 will correspond to projection 410. When device 402is turned an angle β, the portable device ends in position 404. The viewalso turns an angle β while maintaining a camera viewing angle α. As aresult, the view on the portable device corresponds to projection 412.It should be noted that the view on the screen is independent of the eyeposition, such as positions 408 and 406, and the view is independentfrom where the player is. The image on the display depends on theposition of the portable device, which is acting as a virtual camera.

FIG. 5 illustrates how to play an interactive game over a networkconnection, according to one embodiment. Many types of games arepossible within a shared space. For example, the portable device can beused as a paddle to play a game of ping-pong. The device is moved aroundas if it where a paddle that can hit a virtual ball. Players see theball float between the screen and the opponent's screen. In anembodiment of a war game, the player looks through the portable deviceand aims the catapult at the enemies' ramparts. The player pulls thedevice backwards to load the catapult, and then press a button to firethe catapult toward the enemies' castle.

Shared spaces can also be created when players are in differentlocations, as shown in FIG. 5. The players have established a networkconnection to play the game. Each player synchronizes his device to areference point in the player's space, and a virtual reality is created,such as a ping-pong table. The opponent is shown behind his side of thetable, where the movement of an opponent's device is matched to themotions of the opponent's paddle. The game may also add an avatar tohold the paddle, for an even more realistic game experience. Duringplay, each device keeps track of the motion and position of the device.This information is shared with the other device to enable the otherdevice to place a virtual paddle that matches the motion of the device.Other game information is also shared, such as the location and movementof the ball.

FIG. 6 illustrates the process of tapping two portable devices tosynchronize their positions, according to one embodiment. One method forsetting a common virtual or augmented space includes tapping the twoportable devices. Tapping means striking lightly one device against theother. In the scenario shown in FIG. 6, two players are holding portabledevices 604 and 606, respectively. In order to calibrate both devices tothe same reference point, the players tap the devices by placing bothdevices together. In FIG. 6, both devices are placed back to back, butany position is possible for calibration, such as front to back. The keyto detect the tapping is that one or two of the devices notices anabrupt change in motion, such as a sudden deceleration of the device.

For example, both devices can be moving towards each other, and whenthey tap, both devices come to a stop. The inertial modules in thedevices, such as gyroscope 612 and accelerometer 614, notice the changeof momentum and then the tap can be established. In another scenario,one portable device is stationary while the other portable device movestowards the stationary device. When the devices tap, the moving devicewill notice a sudden change in momentum while the stationary device mayor may not detect the tapping as a small change in momentum can beattributed to the natural motion of the player's hand. To detect thetapping, it is enough that one device detects the tapping, and it is nota requirement that both devices detect the tapping simultaneously. Inone embodiment, both devices detect the tapping and if the detection issubstantially simultaneously, then it is determined that the tapping hasoccurred. To determine that the detection if simultaneous, the devicesexchange timing information regarding the event.

In another embodiment, once portable devices 602 and 604 aresynchronized to the same virtual space, their movement is tracked withaccelerometer 614, enabling the creation of a stable and persistentaugmented reality environment regardless of how the portable device ismoved by the user. In yet another embodiment, the inertial movementinformation can be complemented with image data captured with camera 606or 610. The image data can be used to detect the other device andestimate the distance between the devices, as discussed in more detailbelow in reference to FIGS. 10A-12.

FIG. 7 shows two portable devices after tapping, according to oneembodiment. Tapping can be done by having portable devices touch, buttapping can also be done without having both devices actually come incontact. All that is required is that their change in motion occurs atabout the same time. For example, if the fingers of a player gets isbehind a first device, the tapping will be detected when the seconddevice touches the fingers of the player, causing the change in motionof the devices.

Once the tapping is detected, by either device or by both devices, theportable devices exchange data to confirm that the tapping has occurred.For example, the devices can communicate via WiFi or ultrasoniccommunications. A reference point is created, as previously discussed inFIGS. 1-2. The reference point can be situated somewhere in the back ofthe devices, such as the center of each device when the devices tap. Inanother embodiment, the exchange of data can be performed using aTransferJet interface. TransferJet is an interface that enablescommunication when close proximity is detected between two devices.

It should be noted, that the actual reference point for each device maynot be the exact same point in space for both devices. In other words,each device may have a different reference point, although in mostcases, the reference points will be proximate to each other. Theimportant thing is that both devices set up a reference point and thenstart tracking movement around the virtual or augmented space. Theresult is a common virtual space. The reference point can be set in thecenter of the back of the portable device, in the center of the display,where the camera is located, where the accelerometer is located, etc.

Once the reference point is set, the motion tracking modules are resetto zero to set an origin point to measure position in space. Thisoperation is referred to herein as calibrating the portable devices, andthe calculation of the three-dimensional position of the portable deviceis computed in reference to this origin.

FIG. 8 illustrates the creation of a virtual scene around the referencepoint, in accordance with one embodiment of the invention. As the twoportable devices of FIG. 7 are pulled apart, a virtual or augmentedreality play area is created. The virtual objects, such as fighters 804and 806, are assigned corresponding positions in the 3D space where theportable devices are located. The coordinate origin in the 3D space isreference point 808. Because the virtual objects are positioned relativeto a point in space, there is no need for SLAM or ARTAG for maintainingaugmented realities.

The virtual camera associated with the view from the display iscontrolled is by physically moving the portable device around the gameworld, which has been placed in a fixed position in reference to thereal world. It should be noted that the virtual world is not confined tothe space between the portable devices, but can expand and cover areasabove, below, and behind any of the portable devices.

FIG. 9 depicts two players 906 a-906 b in the same space where a virtualreality has been created around reference point 902 on table 904,according to one embodiment. Players 906 a and 906 b have synchronizedtheir devices 908 a and 908 b to common reference point 902, which islocated on top of table 904. Since point P₀ 902 is the reference point,P₀ is also the coordinate origin and it has coordinates (X₀=0, Y₀=0,Z₀=0). The players are inside a room but the virtual reality, alsoreferred to herein as virtual scene, extends beyond the physicalboundaries of the room.

In one example embodiment, the virtual scene is tied to the referencepoint because the geometry of the virtual scene (as seen through ascreen of the device) is based, at least in part, on the referencepoint. For example, the coordinates of the virtual objects in thevirtual scene may be determined with respect to the reference point. Inone embodiment, the reference point is the coordinate origin, thus thereference point has coordinates (0, 0, 0).

The coordinates can be measured using any standard of measure. However,to provide a visual example, and without limitation on actualcoordinates used, if the coordinates of the virtual scene are measuredin meters, an object with coordinates (1, 0, 0) would be situated onemeter to the right of the reference point. Of course, the coordinates ofobjects, real or virtual, may be dynamically updated as the scenechanges, such as when a virtual object moves within the scene. And, thechanges can be defined by actions set by the computer (e.g., interactiveprogram), driven by actions of the user, or combinations of both.Additionally, for sake of clarity, the interactive program can by anytype of program, such as a video game, a business program, an internetinterface, or simply a graphical user interface that provides access todata, to other users, to programs, or to objects that may or may not bedisplayed or projected by a speaker.

Still further, other embodiments may have different coordinates systemsor use scaling. For example, the coordinate system, instead of being aCartesian system, can be polar, spherical, parabolic, etc. Additionally,the reference point does not have to be the origin of the coordinatesystem, and can be positioned at a different place. For sake ofproviding an example, the reference point can be located at coordinates(5, 5, 5) to enable a buffer of 5 meters in each direction before havingto use negative coordinate values in points beyond the 5 meters. Inanother scenario, the virtual objects are built to scale and thecoordinates are also measured in a scale. For example, the virtualobjects may be built on a scale of 1:10, and the geometric axis can alsohave a scale of 1:10, such that an object with coordinates (1, 0, 0) is1 meter away in the “real” world and 10 meters away in the virtualworld.

In FIG. 9, virtual objects include helicopters 914 a-914 c, clouds,birds, sun 916, etc. As players 906 a and 906 b move their devices, theview of the virtual scene changes as if the players were holding acamera into the virtual world. It should be noted that the view shown indevices 908 a and 908 b may include or may not include the referencepoint. For example, device 908 a held by player 906 a is aiming awayfrom reference point 902, therefore reference point 902 is not viewablein device 908 a, although the view in device 908 a is calculated basedon the locations of device 908 a and reference point 902. Further, theactual reference point may be seen in a player's display as some kind ofmarker (such as an “X”) to let the player know where the referencepoint. In other embodiments, the reference point is not viewable and itexists only as a geographical location without any particular markers.

The room includes other static objects besides table 904, such astelevision 912 and window 910. Sun 918 is visible through window 910. Itshould be noted that virtual sun 916 does not have to correspond toactual sun 918. In one embodiment, virtual sun 916 can be placed wherethe actual sun 918 is located, or where a source of light is located inthe room. This way, lighting and shadows in the virtual or augmentedworld will create realistic effects matching the lighting and shadows inthe room.

As seen in FIG. 9, just because the portable devices are synchronized toa point on a table, the virtual objects do not have to be on the tableor near the reference point. The virtual objects can be located anywherein space. When the portable devices include a camera, the staticfeatures in the room can be used by the portable devices to maintain anaccurate measurement of the current position by adjusting their inertialmeasurements with the views from their cameras. Image analysis in theportable device can detect the edges of a window, a light source, theedges of the table, a painting on the wall, a television, etc. Moredetails are described below in reference to FIGS. 13-14.

In one embodiment, the players can do a rapid recalibration of theportable device, by placing the device on the reference point again, andthen entering a command to reset the motion detection modules in thedevice.

FIGS. 10A-10B illustrate the process of detecting a location of anotherportable device using image recognition, according to one embodiment. Ifboth portable devices are the same model and include a camera 1006, itis possible to use camera-captured images to synchronize both devices,as seen in FIG. 10A. For example, portable device 1002 has taken animage with camera 1006. Since portable device 1002 has received acommand to synchronize, portable device 1002 is scanning the image forthe back of the other portable device 1004. An schematic of the imagetaken is shown in FIG. 10B.

Portable device 1022 has detected a rectangle 1010 that matches thecharacteristics of the portable device being searched. Rectangle 1010has horizontal axis 1012 and vertical axis 1014. Horizontal axis 1012 istilted and angle α with respect to horizon 1018. Since portable device1002 knows the dimensions of the other portable device 1004, portabledevice 1002 makes the proper mathematical calculations comparing thesize of the portable device in the image with the real measurements todetermine the distance, location, and orientation of portable device1004.

Once portable device knows the relative position of portable device1004, portable devices 1002 and 1004 exchange location information toset the common reference point and build the common virtual or augmentedreality around the reference point.

FIG. 11 shows how to detect the second device by finding a light sourcein the second device, according to one embodiment. The synchronizationprocess of FIG. 11 is similar to the process described above inreference to FIG. 10, except that portable device 1102 has a lightsource that is detected by the other portable device. When the portabledevices are in the process of synchronizing, the light source is turnedon to facilitate the detection of the light within the image taken withthe camera. The light source can be a Light-Emitting Diode (LED), aninfrared light, a camera flash, etc.

In another embodiment, a depth camera in one portable device is used tomeasure the distance to the other device. Once the distance is known,the reference point can be set based on the position of either device.Additional data, such as image data, can also be used to complement thecalculations of the relative positions of the devices.

It should be noted that it is possible that both devices detect eachother at about the same time. In this case, the calculation of therelative position of the portable devices can be done by one portabledevice or by the other, or it can be a combination of the measurementstaken by both portable devices.

FIG. 12 illustrates detecting the second device by finding the displayof the second device, according to one embodiment. When calibratingportable devices 1206 and 1208 to a common space, one of the devices,such as portable device 1208 is “turned around” such that display 1202is facing the camera in portable device 1206. For example, portabledevice 1208 has shown a message asking the user to “turn device aroundin order have display facing other portable device.”

Display 1202 can then be brightly lit, such as for example showing awhite screen. The brightness of the display facilitates the detection byportable device 1206 of the display. A brightly lit display shows a highcontrast against most backgrounds, including in many cases the front ofthe player holding the device. Other patterns or colors can be added tothe display to improve its detection. For example, FIG. 12 shows acircle with a square pattern inside for easier recognition of thedisplay using geometry.

Other methods for calibrating both devices, also referred to as pairing,can be found in U.S. application Ser. No. 12/647,291 (Attorney DocketNo. SONYP095A) filed Dec. 24, 2009, and entitled “WIRELESS DEVICEPAIRING METHODS”; U.S. application Ser. No. 12/647,296 (Attorney DocketNo. SONYP095B) filed Dec. 24, 2009, and entitled “WIRELESS DEVICEPAIRING AND GROUPING METHODS”; and U.S. application Ser. No. 12/647,299(Attorney Docket No. SONYP095C) filed Dec. 24, 2009, and entitled“WIRELESS DEVICE MULTIMEDIA FEED SWITCHING”, which are incorporatedherein by reference.

In another embodiment, portable device 1208 has two displays, one in thefront and one in the back. In this case, it is not necessary to turnaround portable device 1208 in order to have the display face the otherplayer. Portable device 1206 will be able to detect either the displayin the front or the display in the back from portable device 1208 toperform the calibration.

FIG. 13 shows an embodiment for tracking the portable device via deadreckoning. If the portable device is equipped with a camera facing awayfrom the users face, then it is possible to know the position of thedevice via dead reckoning and inertial navigation. Additionally, thecameras real-world views can be blended with computer generatedgraphics.

Dead reckoning is the process of estimating a current position basedupon a previously determined position, or fix, and advancing thatposition based upon known or estimated speeds over elapsed time andcourse. A disadvantage of dead reckoning is that since new positions arecalculated solely from previous positions, the errors of the process arecumulative, so the error in the position fix grows with time.

FIG. 13 shows two portable devices that have been calibrated to a pointon table 1310. Over time, device 1302A has travelled around the room, asobserved in dead-reckoning measured trajectory 1306. The points intrajectory 1306 show the times where the position was estimated usingthe previous position and the movement of the portable device 1302Asince the last measurement. On the other hand, real trajectory 1308shows the actual trajectory of the portable device in space. Over time,the measured trajectory tends to diverge farther and farther apart fromthe real trajectory due to the nature of dead reckoning and erroraccumulation. However, dead reckoning tracking can be adjusted tocorrect the deviation from the actual trajectory, as described below inreference to FIG. 14.

FIG. 14 illustrates how to adjust dead reckoning using static featuresin the background, according to one embodiment. The dead reckoning driftis usually small, but the accumulation of error over time can createlarger deviations from actual course. In one embodiment, the DR errorsare corrected by video tracking static patterns in the room, such as anedge of table 1301, a window, a television, etc., as previouslydescribed in reference to FIG. 9.

Error correction can be performed at every DR position measurement, orat certain intervals, or every certain number of measurements. Toperform a correction, DR calculates the position of the device and thecamera takes an image in front of the device. Portable device keepstrack of one or more static features and compares the actual location ofthe static feature with the expected location. Since static features donot move, the difference is attributed to DR reckoning error. Theposition is recalculated in order to have the expected static featureand the measured static feature be in the same place.

In another embodiment, the portable devices communicate with each otherand coordinate the motion tracking of both devices by exchanging DR andfeature information. This way a realistic 3D model of the static objectsin the room can be obtained.

FIG. 15 illustrates one embodiment of a calibration method for amulti-player environment. The positional information gained from thedevices sensors (accelerometers, GPS, compass, depth-cam, etc.) istransmitted to other linked devices to enhance the collaborativelymaintained data in the virtual space. In one embodiment for creating acommon shared space synchronized to a common reference point 1502, afirst player 1504A synchronizes her device into the 3D space withrespect to reference point 1502. Other players in the shared spaceestablish a communication link with the first player to exchangeposition and game information. The relative position can be obtain indifferent ways, such as using WiFi triangulation and ping tests todetermine relative positions. In addition, visual information can beused to determine other locations, such as detecting faces of otherplayers and from their faces, possible locations of gaming devices.

In one embodiment, audio triangulation is used to determine relativeposition, by means of ultrasonic communications and directionalmicrophones. Multiple frequencies can be used to perform the audiotriangulation. Once the devices have exchanged position information,wireless communication, such as ultrasonic, WiFi, or Bluetooth, is usedto synchronize the rest of the devices to reference point 1502. Afterall the devices are calibrated, the devices have knowledge of thereference point 1502 and their relative position with respect toreference point 1502. It should be appreciated that other methods can beused to calibrate multiple devices to a shared reference point. Forexample, all devices may calibrate to the same reference point byplacing the device on the reference point in turns.

The virtual scene can be made even more realistic by using shadows andlighting determined by the lighting sources in the room. By using camerafeeds, game environments and characters have scene lighting and shadowsinfluenced by the real world. This means that a player's hand will casta shadow over virtual characters or objects as the hand are reaches intothe virtual world to interact with the virtual objects. Game worldshadows and lighting are adjusted by real world shadows and lighting toget the best effect possible.

FIG. 16 depicts a multi-player virtual reality game, according to oneembodiment. When calibrated positional and image analysis data iscombined with high speed connectivity, positional and game informationcan be exchanged between each of the devices that choose to participatein a shared-space game experience. This allows each player's systemaccess to the camera view and positional information from all otherplayers to synchronize their calibrated positions and share a virtualspace, also referred to as shared space, together.

After players 1602A-1602C have synchronized or calibrated their portabledevices in reference to a point in the common 3D space (such as a pointon a table), the common virtual scene 1604 is created. Each player has aview of the virtual scene 1604 as if the virtual scene, a battle boardgame in this case, were real on a table in front of the players. Theportable devices act as cameras, such that when the player moves thedevice around, the view changes. As a result, the actual view on eachdisplay is independent from the view in other displays and the view isbased only on the relative position of the portable device in respect tothe virtual scene, which is anchored to an actual physical location onthe 3D space.

By utilizing multiple cameras, accelerometers and other mechanicaldevices to determine position, together with high speed communicationbetween portable devices, it is possible to create a 3D motioncapture-like experience allowing players to see, and possibly touch,virtual game characters and environments in believable ways.

Share Space 1604 games utilize devices' high-speed connectivity toexchange information among devices participating in the Share Space gameexperience. The Share Space 1604 play areas are viewed through thedevice by turning the device into a stable “magic window” that persistsin a space between each of the devices. By using a combination of motiontracking, image analysis, and high persistence of information betweeneach device, the play area appears in a stable position even if whendevices move around.

FIG. 17 shows the flow of an algorithm for generating an interactivespace viewable through at least a first and a second device, inaccordance with one embodiment of the invention. The algorithm describesthe process flows being performed on each portable device, and aseparate flow is described for the first device and for the seconddevice. In reference to the first device, the method begins withoperation 702, where the location of the second device is detected or anotification is received that the second device has detected the firstdevice. The method flows to operation 1704 where synchronizationinformation data is exchanged with the second device in order toidentify a reference point in the 3D space. The reference point isrelative to the physical locations of the first and second devices inthe 3D space. Additionally, both devices establish the physical locationin the 3D space of the other device when setting the reference point.

From operation 1704, the method flows to operation 1706 for generatingin the display a view of an interactive scene that includes thereference point and one or more virtual objects. The view shows theperspective of the interactive scene as observed from the currentlocation of the device. After the view is shown in the display, a checkis performed in operation 1708 to determine if the first device hasmoved. If the first device has moved, the method flows to operation 1710to change the perspective for the display according to the new locationof the first device. If the device has not moved, the method loops backto operation 1706 to continue updating the display, such as to reflectchanges of virtual objects during game play.

During operations 1702, 1704, and 1706, the first device can exchangegame and position tracking information with the second device. Thesecond device performs a similar method to the one performed by thefirst device in operations 1712, 1714, 1716, 1718, and 1720.

FIG. 18 illustrates the architecture of a device that may be used toimplement embodiments of the invention. The portable device is acomputing device and include typical modules present in a computingdevice, such as a processor, memory (RAM, ROM, etc.), battery or otherpower source, and permanent storage (such as a hard disk). Communicationmodules allow the portable device to exchange information with otherportable devices, other computers, servers, etc. The communicationmodules include a Universal Serial Bus (USB) connector, a communicationslink (such as Ethernet), ultrasonic communication, Bluetooth, and WiFi.

Input modules include input buttons and sensors, microphone, touchsensitive screen, cameras (front facing, rear facing, depth camera), andcard reader. Other input/output devices, such as a keyboard or a mouse,can also be connected to the portable device via communications link,such as USB or Bluetooth. Output modules include a display (with atouch-sensitive screen), Light-Emitting Diodes (LED), vibro-tactilefeedback, and speakers. Other output devices can also connect to theportable device via the communication modules.

Information from different devices can be used by the Position Module tocalculate the position of the portable device. These modules include amagnetometer, an accelerometer, a gyroscope, a GPS, and a compass.Additionally, the Position Module can analyze sound or image datacaptured with the cameras and the microphone to calculate the position.Further yet, the Position Module can perform tests to determine theposition of the portable device or the position of other devices in thevicinity, such as WiFi ping test or ultrasound tests.

A Virtual Reality Generator creates the virtual or augmented reality, aspreviously described, using the position calculated by the PositionModule. A view generator creates the view that is shown on the display,based on the virtual reality and the position. The view generator canalso produce sounds originated by the Virtual Reality Generator, usingdirectional effects that are applied to a multi-speaker system.

It should be appreciated that the embodiment illustrated in FIG. 21 isan exemplary implementation of a portable device. Other embodiments mayutilize different modules, a subset of the modules, or assign relatedtasks to different modules. The embodiment illustrated in FIG. 21 shouldtherefore not be interpreted to be exclusive or limiting, but ratherexemplary or illustrative.

FIG. 19 is an exemplary illustration of scene A through scene E withrespective user A through user E interacting with game clients 1102 thatare connected to server processing via the internet, in accordance withone embodiment of the present invention. A game client is a device thatallows users to connect to server applications and processing via theinternet. The game client allows users to access and playback onlineentertainment content such as but not limited to games, movies, musicand photos. Additionally, the game client can provide access to onlinecommunications applications such as VOIP, text chat protocols, andemail.

A user interacts with the game client via controller. In someembodiments the controller is a game client specific controller while inother embodiments, the controller can be a keyboard and mousecombination. In one embodiment, the game client is a standalone devicecapable of outputting audio and video signals to create a multimediaenvironment through a monitor/television and associated audio equipment.For example, the game client can be, but is not limited to a thinclient, an internal PCI-express card, an external PCI-express device, anExpressCard device, an internal, external, or wireless USB device, or aFirewire device, etc. In other embodiments, the game client isintegrated with a television or other multimedia device such as a DVR,Blu-Ray player, DVD player or multi-channel receiver.

Within scene A of FIG. 22, user A interacts with a client applicationdisplayed on a monitor 106 using a controller 100 paired with gameclient 1102A. Similarly, within scene B, user B interacts with anotherclient application that is displayed on monitor 106 using a controller100 paired with game client 1102B. Scene C illustrates a view frombehind user C as he looks at a monitor displaying a game and buddy listfrom the game client 1102C. While FIG. 22 shows a single serverprocessing module, in one embodiment, there are multiple serverprocessing modules throughout the world. Each server processing moduleincludes sub-modules for user session control, sharing/communicationlogic, user geo-location, and load balance processing service.Furthermore, a server processing module includes network processing anddistributed storage.

When a game client 1102 connects to a server processing module, usersession control may be used to authenticate the user. An authenticateduser can have associated virtualized distributed storage and virtualizednetwork processing. Examples items that can be stored as part of auser's virtualized distributed storage include purchased media such as,but not limited to games, videos and music etc. Additionally,distributed storage can be used to save game status for multiple games,customized settings for individual games, and general settings for thegame client. In one embodiment, the user geo-location module of theserver processing is used to determine the geographic location of a userand their respective game client. The user's geographic location can beused by both the sharing/communication logic and the load balanceprocessing service to optimize performance based on geographic locationand processing demands of multiple server processing modules.Virtualizing either or both network processing and network storage wouldallow processing tasks from game clients to be dynamically shifted tounderutilized server processing module(s). Thus, load balancing can beused to minimize latency associated with both recall from storage andwith data transmission between server processing modules and gameclients.

The server processing module has instances of server application A andserver application B. The server processing module is able to supportmultiple server applications as indicated by server application X₁ andserver application X₂. In one embodiment, server processing is based oncluster computing architecture that allows multiple processors within acluster to process server applications. In another embodiment, adifferent type of multi-computer processing scheme is applied to processthe server applications. This allows the server processing to be scaledin order to accommodate a larger number of game clients executingmultiple client applications and corresponding server applications.Alternatively, server processing can be scaled to accommodate increasedcomputing demands necessitated by more demanding graphics processing orgame, video compression, or application complexity. In one embodiment,the server processing module performs the majority of the processing viathe server application. This allows relatively expensive components suchas graphics processors, RAM, and general processors to be centrallylocated and reduces to the cost of the game client. Processed serverapplication data is sent back to the corresponding game client via theinternet to be displayed on a monitor.

Scene C illustrates an exemplary application that can be executed by thegame client and server processing module. For example, in one embodimentgame client 1102C allows user C to create and view a buddy list 1120that includes user A, user B, user D and user E. As shown, in scene C,user C is able to see either real time images or avatars of therespective user on monitor 106C. Server processing executes therespective applications of game client 1102C and with the respectivegame clients 1102 of users A, user B, user D and user E. Because theserver processing is aware of the applications being executed by gameclient B, the buddy list for user A can indicate which game user B isplaying. Further still, in one embodiment, user A can view actual ingame video directly from user B. This is enabled by merely sendingprocessed server application data for user B to game client A inaddition to game client B.

In addition to being able to view video from buddies, the communicationapplication can allow real-time communications between buddies. Asapplied to the previous example, this allows user A to provideencouragement or hints while watching real-time video of user B. In oneembodiment two-way real time voice communication is established througha client/server application. In another embodiment, a client/serverapplication enables text chat. In still another embodiment, aclient/server application converts speech to text for display on abuddy's screen.

Scene D and scene E illustrate respective user D and user E interactingwith game consoles 1110D and 1110E respectively. Each game console 1110Dand 1110E are connected to the server processing module and illustrate anetwork where the server processing modules coordinates game play forboth game consoles and game clients.

FIG. 20 illustrates an embodiment of an Information Service Providerarchitecture. Information Service Providers (ISP) 250 delivers amultitude of information services to users 262 geographically dispersedand connected via network 266. An ISP can deliver just one type ofservice, such as stock price updates, or a variety of services such asbroadcast media, news, sports, gaming, etc. Additionally, the servicesoffered by each ISP are dynamic, that is, services can be added or takenaway at any point in time. Thus, the ISP providing a particular type ofservice to a particular individual can change over time. For example, auser may be served by an ISP in near proximity to the user while theuser is in her home town, and the user may be served by a different ISPwhen the user travels to a different city. The home-town ISP willtransfer the required information and data to the new ISP, such that theuser information “follows” the user to the new city making the datacloser to the user and easier to access. In another embodiment, amaster-server relationship may be established between a master ISP,which manages the information for the user, and a server ISP thatinterfaces directly with the user under control from the master ISP. Inother embodiment, the data is transferred from one ISP to another ISP asthe client moves around the world to make the ISP in better position toservice the user be the one that delivers these services.

ISP 250 includes Application Service Provider (ASP) 252, which providescomputer-based services to customers over a network. Software offeredusing an ASP model is also sometimes called on-demand software orsoftware as a service (SaaS). A simple form of providing access to aparticular application program (such as customer relationshipmanagement) is by using a standard protocol such as HTTP. Theapplication software resides on the vendor's system and is accessed byusers through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe Internet. Users do not need to be an expert in the technologyinfrastructure in the “cloud” that supports them. Cloud computing can bedivided in different services, such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).Cloud computing services often provide common business applicationsonline that are accessed from a web browser, while the software and dataare stored on the servers. The term cloud is used as a metaphor for theInternet, based on how the Internet is depicted in computer networkdiagrams and is an abstraction for the complex infrastructure itconceals.

Further, ISP 250 includes a Game Processing Server (GPS) 254 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 256 distributes audio or video signalsto an audience. Broadcasting to a very narrow range of audience issometimes called narrowcasting. The final leg of broadcast distributionis how the signal gets to the listener or viewer, and it may come overthe air as with a radio station or TV station to an antenna andreceiver, or may come through cable TV or cable radio (or “wirelesscable”) via the station or directly from a network. The Internet mayalso bring either radio or TV to the recipient, especially withmulticasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 258 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, a plurality of SSPs can have total or partialcopies of the user data, allowing users to access data in an efficientway independently of where the user is located or the device being usedto access the data. For example, a user can access personal files in thehome computer, as well as in a mobile phone while the user is on themove.

Communications Provider 260 provides connectivity to the users. One kindof Communications Provider is an Internet Service Provider (ISP) whichoffers access to the Internet. The ISP connects its customers using adata transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, wireless ordedicated high-speed interconnects. The Communications Provider can alsoprovide messaging services, such as e-mail, instant messaging, and SMStexting. Another type of Communications Provider is the Network Serviceprovider (NSP) which sells bandwidth or network access by providingdirect backbone access to the Internet. Network service providers mayconsist of telecommunications companies, data carriers, wirelesscommunications providers, Internet service providers, cable televisionoperators offering high-speed Internet access, etc.

Data Exchange 268 interconnects the several modules inside ISP 253 andconnects these modules to users 262 via network 266. Data Exchange 268can cover a small area where all the modules of ISP 250 are in closeproximity, or can cover a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 268 caninclude a fast Gigabit Ethernet (or faster) within a cabinet of a datacenter, or an intercontinental virtual area network (VLAN).

Users 262 access the remote services with client device 264, whichincludes at least a CPU, a display and I/O. The client device can be aPC, a mobile phone, a netbook, a PDA, etc. In one embodiment, ISP 250recognizes the type of device used by the client and adjusts thecommunication method employed. In other cases, client devices use astandard communications method, such as html, to access ISP 250.

Information Service Providers (ISP) 250 delivers a multitude ofinformation services to users 262 geographically dispersed and connectedvia network 266. An ISP can deliver just one type of service, such asstock price updates, or a variety of services such as broadcast media,news, sports, gaming, etc. Additionally, the services offered by eachISP are dynamic, that is, services can be added or taken away at anypoint in time. Thus, the ISP providing a particular type of service to aparticular individual can change over time. For example, a user may beserved by an ISP in near proximity to the user while the user is in herhome town, and the user may be served by a different ISP when the usertravels to a different city. The home-town ISP will transfer therequired information and data to the new ISP, such that the userinformation “follows” the user to the new city making the data closer tothe user and easier to access. In another embodiment, a master-serverrelationship may be established between a master ISP, which manages theinformation for the user, and a server ISP that interfaces directly withthe user under control from the master ISP. In other embodiment, thedata is transferred from one ISP to another ISP as the client movesaround the world to make the ISP in better position to service the userbe the one that delivers these services.

ISP 250 includes Application Service Provider (ASP) 252, which providescomputer-based services to customers over a network. Software offeredusing an ASP model is also sometimes called on-demand software orsoftware as a service (SaaS). A simple form of providing access to aparticular application program (such as customer relationshipmanagement) is by using a standard protocol such as HTTP. Theapplication software resides on the vendor's system and is accessed byusers through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe Internet. Users do not need to be an expert in the technologyinfrastructure in the “cloud” that supports them. Cloud computing can bedivided in different services, such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).Cloud computing services often provide common business applicationsonline that are accessed from a web browser, while the software and dataare stored on the servers. The term cloud is used as a metaphor for theInternet, based on how the Internet is depicted in computer networkdiagrams and is an abstraction for the complex infrastructure itconceals.

Further, ISP 250 includes a Game Processing Server (GPS) 254 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 256 distributes audio or video signalsto an audience. Broadcasting to a very narrow range of audience issometimes called narrowcasting. The final leg of broadcast distributionis how the signal gets to the listener or viewer, and it may come overthe air as with a radio station or TV station to an antenna andreceiver, or may come through cable TV or cable radio (or “wirelesscable”) via the station or directly from a network. The Internet mayalso bring either radio or TV to the recipient, especially withmulticasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 258 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, a plurality of SSPs can have total or partialcopies of the user data, allowing users to access data in an efficientway independently of where the user is located or the device being usedto access the data. For example, a user can access personal files in thehome computer, as well as in a mobile phone while the user is on themove.

Communications Provider 260 provides connectivity to the users. One kindof Communications Provider is an Internet Service Provider (ISP) whichoffers access to the Internet. The ISP connects its customers using adata transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, wireless ordedicated high-speed interconnects. The Communications Provider can alsoprovide messaging services, such as e-mail, instant messaging, and SMStexting. Another type of Communications Provider is the Network Serviceprovider (NSP) which sells bandwidth or network access by providingdirect backbone access to the Internet. Network service providers mayconsist of telecommunications companies, data carriers, wirelesscommunications providers, Internet service providers, cable televisionoperators offering high-speed Internet access, etc.

Data Exchange 268 interconnects the several modules inside ISP 253 andconnects these modules to users 262 via network 266. Data Exchange 268can cover a small area where all the modules of ISP 250 are in closeproximity, or can cover a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 268 caninclude a fast Gigabit Ethernet (or faster) within a cabinet of a datacenter, or an intercontinental virtual area network (VLAN).

Users 262 access the remote services with client device 264, whichincludes at least a CPU, a display and I/O. The client device can be aPC, a mobile phone, a netbook, a PDA, etc. In one embodiment, ISP 250recognizes the type of device used by the client and adjusts thecommunication method employed. In other cases, client devices use astandard communications method, such as html, to access ISP 250.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus may be specially constructedfor the required purpose, such as a special purpose computer. Whendefined as a special purpose computer, the computer can also performother processing, program execution or routines that are not part of thespecial purpose, while still being capable of operating for the specialpurpose. Alternatively, the operations may be processed by a generalpurpose computer selectively activated or configured by one or morecomputer programs stored in the computer memory, cache, or obtained overa network. When data is obtained over a network the data maybe processedby other computers on the network, e.g., a cloud of computing resources.

The embodiments of the present invention can also be defined as amachine that transforms data from one state to another state. Thetransformed data can be saved to storage and then manipulated by aprocessor. The processor thus transforms the data from one thing toanother. Still further, the methods can be processed by one or moremachines or processors that can be connected over a network. Eachmachine can transform data from one state or thing to another, and canalso process data, save data to storage, transmit data over a network,display the result, or communicate the result to another machine.

One or more embodiments of the present invention can also be fabricatedas computer readable code on a computer readable medium. The computerreadable medium is any data storage device that can store data, whichcan be thereafter be read by a computer system. Examples of the computerreadable medium include hard drives, network attached storage (NAS),read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetictapes and other optical and non-optical data storage devices. Thecomputer readable medium can include computer readable tangible mediumdistributed over a network-coupled computer system so that the computerreadable code is stored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

What is claimed is:
 1. A method for generating an interactive spaceviewable through at least a first device and a second device, the methodcomprising: taking an image with a camera in the first device, whereinthe first device and the second device are handheld devices; determininga relative position of the second device with reference to the firstdevice based on image analysis of the image to identify a geometry ofthe second device; identifying a reference point in a three-dimensional(3D) space based on the relative position; and generating views of aninteractive scene in corresponding displays of the first device and thesecond device, wherein the interactive scene is tied to the referencepoint and includes virtual objects, each view showing all or part of theinteractive scene as observed from a current location of thecorresponding device.
 2. The method as recited in claim 1, whereindetermining the relative position further includes: identifying anoutline of the second device in the image.
 3. The method as recited inclaim 2, wherein determining the relative position further includes:identifying a size of the outline; and determining distance to thesecond device based on the size of the outline of the second device. 4.The method as recited in claim 2, wherein determining the relativeposition further includes: identifying a tilt angle of the second devicein the image.
 5. The method as recited in claim 1, further including:sending synchronization information from the first device to the seconddevice after identifying the reference point.
 6. The method as recitedin claim 1, further including: receiving a command to synchronize thefirst device and the second device before taking the image with acamera.
 7. The method as recited in claim 1, wherein generating views ofthe interactive scene further includes: generating a first view on afirst display in the first device presenting the interactive scene asobserved from a point on the first display with a viewing directionperpendicular to the first display, wherein the first view does notdepend on an eye position of a user holding the first device.
 8. Themethod as recited in claim 7, wherein movement of the first device inthe 3D space causes the corresponding view to change according to thecurrent location.
 9. The method as recited in claim 1, whereingenerating views of the interactive scene further includes: generating afirst view on a first display in the first device presenting theinteractive scene as observed from an eye of a user holding the firstdevice, wherein a first display on the first device acts as a windowinto the interactive scene, wherein the first view depends on a positionof the eye of the user.
 10. The method as recited in claim 1, whereinoperations of the method are performed by a computer program whenexecuted by one or more processors, the computer program being embeddedin a non-transitory computer-readable storage medium.
 11. A method forgenerating an interactive space viewable through at least a first deviceand a second device, the method comprising: taking an image with acamera in the first device, wherein the first device and the seconddevice are handheld devices; determining a relative position of thesecond device with reference to the first device based on image analysisof the image to identify a light source in the second device;identifying a reference point in a three-dimensional (3D) space based onthe relative position; and generating views of an interactive scene incorresponding displays of the first device and the second device,wherein the interactive scene is tied to the reference point andincludes virtual objects, each view showing all or part of theinteractive scene as observed from a current location of thecorresponding device.
 12. The method as recited in claim 11, furtherincluding: detecting, in the second device, a command to synchronize thefirst device and the second device; and turning on the light source inthe second device in response to the command to synchronize.
 13. Themethod as recited in claim 11, further including: sendingsynchronization information from the first device to the second deviceafter identifying the reference point.
 14. The method as recited inclaim 11, wherein generating views of the interactive scene furtherincludes: generating a first view on a first display in the first devicepresenting the interactive scene as observed from a point on the firstdisplay with a viewing direction perpendicular to the first display,wherein the first view does not depend on an eye position of a userholding the first device.
 15. The method as recited in claim 11, whereingenerating views of the interactive scene further includes: generating afirst view on a first display in the first device presenting theinteractive scene as observed from an eye of a user holding the firstdevice, wherein a first display on the first device acts as a windowinto the interactive scene, wherein the first view depends on a positionof the eye of the user.
 16. A method for generating an interactive spaceviewable through at least a first device and a second device, the methodcomprising: taking an image with a camera in the first device, whereinthe first device and the second device are handheld devices; determininga relative position of the second device with reference to the firstdevice based on image analysis of the image to identify a graphic shownon a second display of the second device; identifying a reference pointin a three-dimensional (3D) space based on the relative position; andgenerating views of an interactive scene in corresponding displays ofthe first device and the second device, wherein the interactive scene istied to the reference point and includes virtual objects, each viewshowing all or part of the interactive scene as observed from a currentlocation of the corresponding device.
 17. The method as recited in claim16, further including: prompting a user holding the second device,before taking the image, to place the second display in a positionfacing the camera in the first device.
 18. The method as recited inclaim 16, wherein the graphic shown on the second display includes abrightly lit second display.
 19. The method as recited in claim 16,wherein the graphic shown on the second display includes a geometricfigure.
 20. The method as recited in claim 16, wherein operations of themethod are performed by a computer program when executed by one or moreprocessors, the computer program being embedded in a non-transitorycomputer-readable storage medium.